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SUMWRY 



Problem 

The Advanced Instructional System (AIS) has the objective of 
testing and evaluating the feasibility of Urge scale, computer-based 
Instructional systems for Air Force training. This project investi- 
gated one of the AIS Courses, 3 ABR 32430-2, Precision Measuring 
Equipment (PME) Specialist. The specific objectives of this project 
were to demonstrate the feasibility of individualizing the PME course 
as part of the AIS, to provide -instructional material development 
guidelines for use in the AIS, and to produce prototype materials 
adaptable for use in the AIS. 

Approach 

The approach to the problem is best described in terms of the four 
phases of its accomplishment. The Phase I steps were: delineation of 
criteria for selecting course segments for individualization, analysis 
of the PME course and selection of representative se^nts for indivi- 
dualization. The Phase II steps vi«re analysis of selected se^nts, 
and selection of strategies and Instructional media. Phase III steps 
were design and preparation of Individualized instructional materials, 
instructional management procedures, and a computer adaptation rationale. 
iThe Phase IV steps were tryout and evaluat16n"of tne prototype materials. 

Results 

The learning materials were tried out on the then currently avail- 
able PME classes. The materials for a 90-hour block of instruction 
included printed and audiovisual I7«dia and covered complex cognitive 
and performance skills. For this block there was a 31% reduction in 
mean training time with all trainees achieving ell criterion objectives. 
Trainees achieved 99% of the performance objectives on the first attempt. 
The mean written test score was 80.1% (the passing score is 70%). Three 
of 35 trainees required remediation because of low written test scores. 
Trainee attitudes toward the materials and system as measured by an 
attitude scale were positive. The attitude scale has a range of 20-100 
with 60 as neutral. The mean trainee attitude scale value was 78.4. 
Three smaller segments of instruction were also tried out. The materi- 
al', were printed adjunct programs and covered cognitive skills. The 
normal group-paced times were 6, 6, and 9 hours- The reduction in 
training times were 60%, 70%, and 61%, respectively. The written test 
scores were 83%, 96%, and 99%. There were 5.615 man-hours of direct 
labor expended in developing the Block X materials and the smaller 
segments (111 hours of conventional instruction in dll). The mean 
hours of effort expended per trainee contact hour was 79. Production 
costs for the Block X instructional materials (software and hardware) 
were $24,327.00. 
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Conclusions 



The reduction in training time coupled with trainee achievewent 
supports the feasibility of Individualized PME as part of the AIS. 
No problems were encountered that would indicate otherwise. The 
design and development of the learning materials required close coor- 
dination with instructional staff experts. Face-to-face contact was 
required to preclude misunderstandings and collocation of personnel 
is recorrmended where possible. It is also desirable that the staff 
personnel involved in the development of instructional materials are 
also part of the staff used to try out the materials. Because the 
management of individualized instruction is not trivial. It Is impor- 
tant that the new instructor roles be defined early and the Instruc- 
tors be trained to accomplish these roles. Even with this pre- 
planning and training, a certain evolutionary change In roles takes 
place. During the tryouts it became necessary to provide a separate 
testing area for trainees taking written tests. The testing and 
trainee evaluation process has greater space and personnel require- 
ments during tryouts than are needed after materials are formally 
implen^nted. It was also determined that equi^nt spares of all 
types are required to support Individualized training. 

Estimating the cost effectiveness of this type of effort can be 
done by making certain assun^tlons. The contract costs are known 
and trainee flows are defined in Air Force training flow charts. 
There are several available estinates of course costs. Using a two- 
year time period and a minlmimi training cost estimate (that used for 
the Air Force Resources Conservation Program), the projected savings 
for 1,025 students of reduced course length resulting from the con- 
version of Block X into self-paced instruction would be $188,457.53. 
Subtracting total contract costs and allowing a more than adequate 
maintenance allowance, a minimum savings of $40,000.00 per year would 
be realized. If trainee pay and allowances for the time saved were 
added, the savings would increase to $110,000.00 per year. Such 
projected savings for one block of instruction argue strongly for 
the cost effectiveness of efforts such as this. 
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SECTION I 
INTRODUCTION 

The Advanced Instructional System (AIS) Is Project 1193 of PE 
63102F, "Innovations in Training and Education, (INNOVATE)." The 
overall objective of Project 1193 is to design, develop, test and 
evaluate the feasibility of a potentially large-sca'e, conputpr- 
based instructional system for Air Force training and education. 
The AIS will be an integrated training system encompassing indivi- 
dualized, self-paced course materials, a variety of instructional 
media, and a coriputer training management subsystem. 

Several technical training courses will be implemented under the 
Initial AIS configuration. Among them is 3ABR32430-2, Precision 
Measuring EquijHnent (PME) Specialist, a course that involves complex 
cognitive concepts and electronic skills. Airmen are taught to main- 
tain, calibrate, troubleshoot, and repair sophisticated equipment In 
laboratories whose standards are one level below thost* of the National 
Bureau of Standards. The work described in this report constitutes 
the initial effort in the development of the PME course as part of 
the AIS. 

The specific tasks that we had to accanplish included developing 
prototype Individualized instructional materials for se^nts of the 
Pl€ course, and docun«nting the procedures utilized to provide Indi- 
vidualization. The overall objectives were to: 

a. demonstrate the ffcjsibillty of individualizing the PME 
course as part of the AIS; 

b. provide instructional materials development guidelines for 
use in subsequent AIS efforts; and, 

c. produce individualized Instructional materials amenable to 
adaptation for eventual use as part of the integrateo AIS. 

Our approach for realizing these objectives consisted of fotr 
major groups of activities. These groups of activities and the o-der 
in which they were accomplished are as follws: 

Phase I. Delineation of criteria for selecting course 

segments for individualization, analysis of the 
PME course and selection of representative 
segments for individualization; 

Phase II. analysis of designated segments and selection of 
strategies and 1i structional media; 
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Phase III. design and preparation of Individualized Instruc- 
tional materials, instructUnal management pro- 
cedures, and conputer adaptation rationale; and, 

Phase IV. tryout and evaluation of the materials. 

In the ensuing sections, the specific tasks that compromised each 
major group of activities are discussed in detail. Taken together, 
these tasks and the guidelines resented for accomplishing them 
constitute a model for the selective development and evaluation of 
self-paced, rmjltimedia technical training materials. 



DELINEATION OF CRITERIA FOR SELECTING COURSE 
SEGMENTS FOR INDIVIDUALIZATI(»<, ANALYSIS 

OF THE COURSE, AND SELECTION OF 
SmU SE01ENTS FOR INDIVIDUALIZATION 

The tasks that comprised each of the three major activities in 
Phase I and the order in which they mre carried out was as follows: 

ACTIVITIES TASKS 

Delineation of Criteria for 1. Develop Selection 



SECTION II 



Selecting Course Segments 
for Individualization 



Criteria Based on 
Study Objectives, 




2. Gather, Review, and 
Analyze Relevant 
Course and Job 
Documents. 



Analysis of the Course 



3. Isolate Important 

Trainee Characteristics. 



4. Examine the Training 
Enviroranent. 



Selection of Sample Segments 
for Individualization 



5. Select Sample Segments 
for Individualization on 
the Basis of Information 
Gathered in Tasks 1 
through 4. 
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Detailed descriptions of how these tasks were acconplished are 
presented in the pages that follow. 

DELINEATION OF CRITERIA FOR SELECTING COURSE 
SEGMENTS FOR INDIVIDUALIZATION 

TASK 1 : DEVELOP SEGMENT SELECTION CRITERIA 

In keeping with our objective of c«»<Tionstrating the feasibil- 
ity of Individualizing the PME course as part of the AIS, we 
developed the following criteria for selecting course segments 
for individualization: 

1. Segnents chosen mst sample the full range of skills 
and knowledges required in the PME course. 

2. Secpnents chosen must sample the full range of media 
requirements for the PME course. 

3. Segments chosen must contain skills and knowledges 
which are prerequisities for accomplishing high fre- 
quency tasks on the job. 

4. Segments chosen must contain skills and knowledges 
which are essential to projected as well as current 
high frequency job tasks. 

5. Se^ents chosen must be able to be implemented without 
disrupting trainee flow through nonselected segments. 

Having defined the selection criteria, our next task consisted 
of analyzing the PME course so that segments meeting these cri- 
teria could be identified. 

ANALYSIS OF THE COURSE 

TASK 2: GATHER. REVIEW, AND ANALYZE RELEVANT COURSE AND 
JOB DOCUMENTS 

The first step in the course analysis process involved 
gathering PME course and job documents. Our preliminary inven- 
tory included the following types of documents: 

0 Course Chart 

0 Specialty Training Standard (STS) 
0 Plan of Instruction (POI) 



0 student Study Guides 



0 Student Workbooks 

0 Occupational Survey Report (OSR) 

0 Progranmed Texts 

0 Student Handouts 

Of these documents, the Course Chart, the POI, the Student 
Study Gdldes and Workbooks, the STS, and the OSR proved nwst use- 
ful in providing an overview of the PMF course. The Course Chart 
contained a description of each block in the course in terms of 
the specific topics covered and the anount of classroan time 
devoted to each topic. The POI was used to identify the critical 
intennediate and terminal behaviors in each block of the course, 
as well as the media and reference materials currently utilized. 
The Student Study Guides and Workbooks gave us a feel for the 
hierarchical relationships that exist within and between blocks, 
and a general idea of the strategies currently enployed In con- 
structing printed materials. The STS provided specific profi- 
ciency requirements for all terminal and Intemwdlate behaviors. 
Finally, task, frequency of j^rformance, and total time data from 
the OSR were utilized in a preliminary determination of those 
terminal and Intermediate behaviors listed in the POI that were 
important from the standpoint of performance on the job. 

TASK 3: ISO-ATE IMPORTANT TRAINEE CHARACTERISTICS 

The second step in the course analysis process was based on 
the assumption that a primary consideration in designing instruc- 
tion is who the program is for and what these trainees wil'i be 
like when they start the program. The characteristics of the tar- 
get population have implications for virtually all aspects of 
instructional design and, for this reason, it was important to 
"define" the target population at an early stage in the design 
activities. 

The specific characteristics of the target group, including 
the degree of honK)geneity or heterogeneity, provide Information 
needed to most effectively make decisions iri such areas as the 
following: 

1. Knowledges and skills to exclude because the target group 
possesses them. 

2. Entry competencies to plan for in knowledge-skill areas 
pertinent to the program. 
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3. Probablt areas for prerequisite deficiencies. 

4. Attitudes to contend with, positive and neqative, includ- 
ing level of achievanent motivation. 

5. Probable differences in learning rates or amounts of 
learning per unit of time. 

6. Appropriateness of different instructional strategies, 
including medii applications. 

The iiT9)ortance of such information for decision-making was 
fully understood in the PME project. Accordingly, steps were 
taken to obtain "readily available" information regarding the 
target population for the new individualized, imiltlmedla PME 
course segnents at Lowry Air Force Base. 

Tbe following types of information about the target popula- 
tion were thought to be Important: 

1. Size and kocation 

a- Size of target population and rate df Increase or 
decrease. 

b. Location of the group. 

c. The instructional "unit" size (e.g., 120 trainees, 
12 per class and 10 classes), If one currently 
exists. 

2. Aptitudes and Achievement 

a. Percentile scores on indices of the Airman Qualify- 
ing Examination (AQE) relevant to the qeneral and 
specific abilities demanded by the FME course. 

b. Scores (mean, range, and standard deviation) on 
standardized national and/or validated local achieve- 
n^nt tests. 

Information regarding the size and location of the target popula- 
tion and their percentile scores on indices of the AQE relevant to 
the PME course was supplted by the instructors at the PME school. 

Initially, the available information was utilized to get an 
idea of how many trainees would interact with the inidividualized 
materials at any one time in each course block, and som feel for 
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the skin and aptitude ranges of these groups. Eventually, this 
infcmiation was utilized to make detailed materials design deci- 
sions with regard to assumed entry skills or deficiencies, learn- 
ing rates, and the appropriateness of various instructional strat- 
gies. 

TASK 4: EXAMINE THE TRAINING ENVIRONf€NT 

The third step in the course analysis process involved sur- 
veying the classroom facilities and equipment inventory at the 
PME laboratory. There were twelve classrooms, one each for Blocks 
I through X, and two for Block XI. The rooms ranged In size from 
700 sq. ft. to 1200 sq. ft., and all were equipped with 117 volts 
AC, 60 Hz wiring. Classrooms devoted to academic topics (the 
initial course blocks) contained tables and chairs. Those class- 
rooms devoted to "hands-on" performance contained workbenches in 
addition to tables and chairs. In both types of classrooms, there 
was ample workspace for individual utilization of printed and 
audiovisual instructional materials. 

Calibration standards and applied measurenent trainers are 
the major equipment iteflis utilized in the PME course. There !ii«re 
limited numbers of some pieces of equipment; however, trainees 
typically worked individually or in pairs in those blocks where 
such equipment was utilized. Our brief survey indicated that most 
course segments could be individualized without disrupting trainee 
flow through nonselected segments or drastically altering the 
physical environment. 

The review and analysis of course and Job documents, trainee 
characteristics, and the training environment gave us a g(M}d feel 
for the broad range of skills and knowledges that the PME graduate 
must possess, an idea of the number and caliber of trainees that 
were entering the course, and a preliminary list of training 
environment resources and constraints. This information was 
utilized in conjunction with the criteria developed in Task 1 to 
select course segments for individualization. 

SELECTION OF REPRESENTATIVE 
SEGMENTS FOR INDIVIDUALIZATION 

TASK 5: SELECT SEGMENTS FOR INDIVIDUALIZATION 

Air Training Cofrmiand (ATC) specified that Block X (Waveform 
Analysis) of the PME course be converted into individualized 
instructional sequences. This block represents gO hours of 
instruction, much of which is of the "hands-on" variety. Inter- 
views with PME Master Instructors indicated that Block X satisfied 
selection criteria 3 and 4 (i.e., ^.he skills and knowledges con- 
tained in the block were essential to current and projected high 
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frequency job tasks). Our analysis Indicated that Block X also 
satisfied selection criteria 2 and 5 (i.e., media requirements 
were representative and trainee flow through nonselected segments 
would not be hindered). To insure that the full range of skills 
and knowledges required in the course were represented {selection 
criterion 1), we selected an additional 24 hours of instruction 
from Blcoks IV and VI that were primarily academic (i.e., theoret- 
ical) in nature. The specific topics chosen 1n thpse blocks were: 
Block IV - Semiconductor Physics (6 hours) and NPN and PNP Transis- 
tors (6 hours); and Block VI - Logic Circuits (9 hours). Pf€ 
Master Instructors reviewed and approved these selections. 

The three lessons from Blocks IV and VI dealt exclusively with 
knowledges and contained no performance tasks. They were used 
primarily as a test-bed for our programmed Instruction technique. 
Consequently, the design, development, and evaluation of Block X 
materials Is emphasized in this report. 

SECTION III 

TRAINING ANALYSIS AND MEDIA SELECTION 

The tasks that comprised the two major activities in Phase II 
and the order in which they were carried out was as follows: 

ACTIVITIES TASKS 

(Pnase I) (U 2. 3, 4, 5) 

Conduct Training Task 
Analysis 

Review Learning Objectives 

Examine ond Update Test 
Items 

Specify Instructional 
Sequence 

Specify Presentation 
Strategies 

Develop Rationale for 
Selecting Media 

Select Media 
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Detailed descriotions of how these tasks were accomplished are pre- 

sei.'ted in the paragraphs that follow. 

DETAILED ANALYSIS 

TASK 6: CONDUCT TRAINING TASK ANALYSIS 

The first 5tep in the detailed analysis of selected segments 
involved -^viewing the POI and STS to identify critical skills and 
knowledges. Block X was comprised of five major segments. For 
each segment, instructional topics were listed. For each topic, 
a terminal behavior or list of terminal behaviors was identified. 
These terniinal behaviors comprise the tasks that the trainee has 
to be able to perform, A list of content stateii«nts was also 
generated for each topic. These content statements describe. In 
narrative form, what the trainee has to know In order to perform 
the tasks. 

Once required tasks and contents had been identified detailed 
task and content analyses were compiled. Each task was broken 
down into the distinctly different steps that comprised it. Each 
step was put in sentence form and the steps were ordered sequen- 
tially. The set of steps then constituted a procedure which could 
be used to perform a "hands-on" task. Task analysis data were 
generated by viewing videotapes of experts performing the tasks in 
question, observing instructors in "live" demonstrations, and 
reviewing technical manuals that dealt with task perfomance. 

To identify critical content, we asked the following question: 
"What would the trainee have to know In order to perform the task 
in question?" Content analyses were written in the same manner 
that conventional subject matter outlines are written. Titles 
and subtitles of one or few words were utilized, with the titles 
and subtitles arranged in a sequence that corresponded to the 
order in which they were to be taught. Content analysis data were 
taken from the existing POI, and the tests and training materials 
referenced In it. The content ultimately included in the materials 
was selected in light of the tasks that the trainee had to perform 
and the terminal behaviors that he had to exhibit. Only informa- 
tion (i.e., content) that supported task performance was Included. 

Preliminary training task analysis results were reviewed with 
Master Instructors from the PME school. Detailed task and content 
data were revised and updated to insure accuracy and completeness. 
Samiile task and content analyses are shown in Appendix A. 
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TASK 7: REVIEW AND UPDATE LEARNING OBJECTIVEI. 

The second step In the detailed analysis of selected se^nts 
involved reviewing and updating criterion and enabling objectives/ 
teaching steps on the basis of task and content analyses. Exist- 
ing objectives were reviewed to isolate the following type* of 
deficiencies: 

0 Inadequacy of form (i.e., objective does not clearly 
state the behavior required, the conditions, or the 
standards of task performance). 

0 Inadequacy of content (i.e.. objective is too broad or 
too narrow to reflect the task which it is to cover). 

Objectives were modified if they were Inadequate in form or in 
content, or deleted if they were unnecessary. Modifications and 
deletions were done under the cognizance of the PME Master Instructors - 

Where required, new objectives were written. The questions 
asked in the process of deciding whether or not to create a cri- 
terion objective for a specific task were as follows: 

0 Is the performance critical and con^Jletely self-contained? 
If yes, answer question below. 

0 Is the performance complex enough that it warrants its own 
objective? If yes. write a new criterion objective. 

The guideline used in deciding if a criterion objective required 
enabling objectives/teaching steps was as follows: 

0 Does the criterion objective as stated assume some pre- 
requisite skill or knowledge for terminal performance? 
If yes. write a new enabling objective/ teaching step. 

New criterion and enabling objectives/ teaching steps were reviewed 
and approved by the PME Master Instructors. A sample learning 
objective is shown in Appendix B. 

TASK 8: EXAMINE AND UPDATE TEST ITEMS 

The third step in the detailed analysis of selected segnents 
involved reviewing and updating test items. We decided that a 
criterion test item had to be essentially a "mirror image" of the 
behavioral objective for which it was prepared and it had to be an 
objective measure. In other words, the exact criterion behavior 
specified in the behavioral objective was to actually be a part 
of the learning objectives. Every criterion behavior stated in 
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the objective had to be explicit, measurable, feasible, unantlguous, 
etc., because that behavior, perhaps exactly as stated, was the 

"answer" to the associated test item. 

The primary advantage of this "mirror Image" approach Involved 
the content validity of the test Items. It allowed us to determine 
whether or not the test Itans were measuring the same behaviors as 
called for by the behavioral objectives. This approach enabled 
"face validity" judgments to be unequivocally made by instructional 
designers. Existing test items were reviewed and updated, and 
some new test items were created for use as embedded questions or 
as supplements to existing examinations. The updated and new test 
items were reviewed and approved by the PME Master Instructors. 

Performance and written tests were constructed for use In 
evaluating trainee attainn»nt of criterion behaviors. The perfor- 
mance tests or criterion- reference checKs were to be administered 
at the end of lessons. and consisted of having the trainee demon- 
strate proficiency on some training task, such as "calibrating the 
545." Trainees had to denronstrate complete mastery of the task in 
question before being allowed to continue. 

Written tests were of two types. One type, called "demand- 
response", consisted of a series of short questions en^edded within 
the PI text. These questions dealt with Important information, 
and had to be answered correctly before the trainee could continue. 
The other type of written test was to be administered when all 
lessons were completed and was con?)osed of test items that, as 
closely as possible, reflected criterion performance. This test 
was called the end-of-block examination. 

TASK 9: SPECIFY INSTRUCTIONAL SEQUENCE 

The fourth step in the detailed analysis of selected se^nts 
involved determining the sequence of instructional activities, 
that is, the order in which trainees would Interact with units of 
content. The existing POI proved to be an effective first step in 
sequencing the instruction. It provided the initial, "general" 
sequence of the topics that comprised Block X. 

An effective second step in sequencing was the preparation of 
training task analysis lists and learning objectives. Identifying 
training taskr, preparing criterion and enabling objectives, and 
examining the n lationships among criterion objectives and their 
associated enabling objectives provided the information necessary 
to sequence the block into a hierarchy of competencies. Block X 
was divided 1nt» the rollnwinn five major segments: 
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IV 

Calibration of 
Osclllosco^s 
and Oscilloscope 
Calibrating Equipment 



1 



II 

Circuit Analysis 
and Maintenance 
of Oscilloscopes 
and Preamplifiers 



III 

Circuit Analysis 
and Maintenance 
of Oscilloscope 
Calibrating Equipment 



Use and Block 
Diagram Analysis 
of Standard 
Sa'npling System 



I 

Uses of 
Oscilloscopes 
and 

Preamplifiers 



FIGURE 1. Major groupings of objectives. 
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0 Uses of Oscilloscopes ;.nd Preamplifiers 

0 circuit Analysis and f^lntonance of Oscilloscopes and 
Preampl ifiers 

0 Circuit Analysis and i^alntenance of Oscilloscope Calibrat- 
ing Equipment 

0 Calibration of the Oscilloscope and Oscilloscope Calibrat- 
ing Equipment 

0 Use and Block Diagram Analysis of Standard San^llng System 

Each of these segments was, In turn, then broken down Into sections 
(lessons). A section or lesson was a self-standing unit of 
instruction that dealt with a criterion objective or group of 
closely related criterion objectives. 

The five major groups of objectives represented the five irwjor 
concept areas or segments of Block X and were related in the fashion 
illustrated in Figure 1. Note that the se^nents are hierarchically 
ordered. It was necessary for a trainee to achieve the objectives 
of Segment I before he co^ild begin either Segment 11 or Segment III 
or Se^nent V. Similarly It was thought that the trainee had to 
have achieved the objectives of Segpnents II, III. and V before he 
was qualified to begin Se^nt IV. 

In terms of course presentation, then, a trainee could study 
the segments in the orders of sequences sh'3wn In Figure 2. The 
decision to allow all acceptable sequences was based on the fact 
that limited nunters of test and calibration equ1pn«nts would be 
available during evaluatinn of the prototype Instructional materiaU. 
Additionally, there were alternative lesson sequences for certain 
of the segments. A flowchart showing acceptable lesson sequences 
appears In Appendix C. Multiple routes through the materials con- 
stituted a mechanism for maintaining trainee flow and minimizing 
training time in the face of limited equipment Inventory. 
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TASK 10: SPECIFY PRESENTATION STRATEGIES 

The fifth and last step In the detailed analysis of segments 
involved specifying presentation strategies. Since one of our 
major objectives was to test the feasibility of Indi vidua! l2i no 
a portion of the PME course, we decided that each trainee should 
largely control his own learning pace, provided he Is progressing 
at some minimally acceptable rate, and that a variety of Instruc- 
tional media should be provided. In order to facilitats self- 
pacing, the instructional materials wre to be prepared In such 
a way that each trainee could control, within Mmlts, the amount 
of time that he spent interacting with each set of lesson materials. 
Additionally, we decided that the nwterials were to be prepared in 
such a w4y that each trainee would: 

a. receive instruction that was "structured" to provide 
information regarding the skills and knowledges to be 
mastered, 

b. have the opportunity to actively practice the skills and 
knowledges to be mastered, and 

c. receive feedback regarding the adequacy of his perfor- 
mance during practice. 

The impact of these strategies, decisions on the selection of 
Instructional media and the construction of Instructional materials 
are detailed in the paragraphs that follow. 

MEDIA SELECTION 

TASK 11: DEVELOP RATIONALE FOR SELECTING MEDIA 

Three factors were considered In choosing instructional media 
for Block X lessons. The first factor was the type of learning 
that the lesson objectlve(s) entailed. The second factor was the 
necessity (and desirability) of producing self-paced and audio- 
visual (AV) materials. The third factor was the constraints under 
which the materials were being developed. Consideration of each 
of these factors was formally represented as one stage In a 
three-stage media selection process (cf. Figure 3). 

Stage 1 Involved classifying the type of learning entailed in 
the lesson objectlvejs) as either the acquisition of a knowledge 
or the acquisition of a skill. Two decisions were then made. One 
was that some form of self-paced progrsnmed Instruction was a suit- 
able vehicle for imparting knowledges of the type required in the 
Pf€ course. The other was that some form of Instruction which 
combined demonstration and performance was a suitable vehicle for 
Imparting skills of the type •squired in the Pl€ course. 
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FIGURE 3. Three-stage media selection process. 
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Stage 2 involved Identifying candidate media «nd media mixes 
that could be used to Implement self-paced programmed Instruction 
for "knowledge" lessons and demonstratlon/performar ce-based 
instruction for "skill" lessons. For example, sell -paced progranmed 
instruction could have beeri implemented using printed materials 
exclusively, audio tapes and printed workbooks, or microfiche and 
chemically-treated answer sheets. Dewnstration/pf rformance-based 
instruction could have beefi Implemented using varicus AV media, 
either singularly or in ccsnbi nation, live demonstrations and super- 
vised practice, or computer-assisted instruction. 

Stage 3 involved defining the constraints that would Impact 
media selection. Learning proficiency and cost were the primary 
constraints considered in the choice of a median fcr In^lementing 
self-paced programmed instruction for "knowledge" lessons. The 
slection of the n^ium was made with a view toward individualizing 
instruction and maximizing learning, while minimizing material 
production, reproduction, and revision costs. Two constraints 
Influenced the choice of n*>d1a used to implement the demonstration/ 
perfoiTTiance paradisKn for "skUI" lessons. The first constraint 
arose as a result of our initial decision to produce self-paced 
progranmed instruction for "knowledge" lessons. We assumed that 
the trainees would progress through the self-paced materials at 
different rates and we abandoned the traditional notion of the 
Instructor confronting the group with "live" demonstrations for 
"skill" lessons. This argued for "canned" demonstrations that 
could be made available to trainees on an individual basis, and 
dictated our choice of AV media for "skill" lessons. The second 
constraint, the desirability of utilizing AV media that were com 
petible with presentation hardware of demonstrated reliability 
already in the Air Force inventory, helped narrow the range of AV 
media under consideration. 

TASK 12: SELECT MEDIA 

The primary medium selected for "knowledge" lessons was pro- 
grammed text. The primary nedia chosen for "skill" lessons were 
filmstrlp/sound and motion picture programs. Supplementary media 
included videotape programs, AV scripts, and a picture book con- 
taining annotated equipn^nt photographs. 

SECTION IV 

PREPARATION OF MATERIALS, DEVELOP^T OF INSTRUCTIONAL 
MANAGEMENT PROCEDURES, AND QlSW\ OF A COMPUTER ADAPTATION RATIONALE 

The tasks that comprised the major activity in Phase III and 
the order In which they were carried out was as follows: 
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ACTIVITIES 



TASKS 



(Phase I) - 



(Phase II) - 



(1. 2. 3. 4. 5) 

(6. 7. 3. 9. 10. 11, 12) 




Prepare Instructional 
Materials 



Instructional Materials 
Development 



Develop Instructional 
Management Procedures 



Delineate Ct^uter 
Adaptation Rationale 



Detailed descriptions of how these tasks were acconplished are 
presented in the pages that follow. 

TASK 13: PREPARE INSTRUCTIONAL MATERIALS 

A programned text was (teveloped for each of the five se^nts 
that comprised Block X. Each text consisted of prograiwied lessons, 
with each lesson containing all of the information necessary to 
master one of the PME knowledges identified during the training 
analysis. Lessons were based on the content analyses conpleted in 
Phase II, and were narrative in format. Underlining and embedded 
questions were used extensively. IMfiderlining critical content 
within lessons provided "structured" instruction in the sense that 
knowledges to be mastered were hi^lighted and emphasized. Questions 
were embedded within lessons to give trainees the opportunity to 
actively practice the knowledges to be mastered. The answers to 
the embedded questions were placed at the end of each lesson to 
provide feedback regarding the adequacy of performance during 
practi ce. 

Sound/fi Imstrip (Audiscan) programs were prepare'^ for the bulk 
of the "skill" lessons. Using the task analyses compseted in 
Phase II, scripts were written outlining every distinct step listed 
in each task analysis. For each of these steps, a scene was staged 
with .' J appropriate test equipment and, in some cases, an actor. 
Each scene wat> then photographed; and 35W color sl1(fe$ produced 
and placed in an order that corresponded to the sequence In which 
the steps had to be performed. The slides and script were used, along 
with the appropriate test equipment, in a "dry run" of the procedure. 
PME Master Instructors observed the "dry run," recommended changes 
where appropriate, and examined the revised product prior to certifying 
the technical accuracy of the slides and script. Each slide package 
was rephotographed as a 16W1 filmstrip. The script was narrated onto 
1/4 inch magnetic tape and used as a master. Copies were made of both 
the filmstrip and the narrative and the copies were loacted into Audiscan 
cartridges. 
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The sound/filmstrip programs provided "structured" instruction 
in the sense that each PME task was fully proceduralized; i.e., 
each step was clearly and concisely presented. Using actual PME 
equipment, trainees performed each step in each procedure by Imi- 
tating the actions seen on the screen, thereby actively practicing 
the skills to be learned. Feedback slides were included in each 
program to provide a model or standard against which trainees 
could periodically assess the adequacy of their responses. 

Motion pictures (A. B. Dick programs) were produced for those 
"skill" lessons that Involved, as an integral component of instruc- 
tion, the recognition of systematic changes In dynamic display pat- 
terns In response to specific equipment adjustments. Using the 
task analyses completed in Phase II, scripts were written detailing 
each step in each task. For each of these steps a scene was staged 
with the appropriate test equipment. A 16MM motion picture camera 
and color film were used to photograph an actor making th^ required 
equipment adjustments as well as the dynamic scope patterns that 
resulted from those adjustn«nts. The film was edited, and used, 
along with the script, in a "dry run" of the procedure. PME Master 
Instructors observed the "dry run", reconwended changes where 
appropriate, and examined the revised product prior to certifying 
the technical accuracy of the film and script. A "narrative-over- 
picture" procedure Insured proper synchronization of the master. 
Super 8W1 magnetic strip reduction prints were made and loaded 
into A. B. Dick cartridges. The A. B. Dick programs provided for 
'structured" instruction, active practice, and feedback in a manner 
essentially identical to t"at of the Audiscan (sound/filmstrip) 
programs . 

A book of equipment Photographs was also compiled. Each 
piece of PME equipment was photographed, along with those specific 
areas of each piece that contained calibration or test points, 
with high-resolution color film. The films were developed and 
enlarged to 8 1/2 x 11 color prints. Reference lines and numbers 
were superimposed on those prints containing calibration or test 
points by placing acetate overlays on the original prints and 
rephotographing them. These films were processed and enlarged to 
8 1/2x11 color prints. Written descriptions of calibration or 
test points were referenced to the annotated photos. Trainees 
were to use the picture book to familiarize themselves with Block 
X equipments. Additionally, the picture book could be used in 
conjunction with Audiscan and A. B. Dick scripts to preview AV 
sequences. 

Finally, each Block X procedure being performed by a PME 
Master Instructor was videotaped. The start/stop and rewind 
features of the videotape presentation devices made it possible 
fo;" trainees to preview an AV sequence or to selectively review 
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a portion or portions of an AV sequence as often as desired and on 
a task step-by-step basis if necessary. 

TASK 14: DEVELOP INSTRUCTIONAL MANAGEMENT PROCEDURES 

The pre-AIS impleti^ntation of Block X in an individualized, 
self-paced mode necessitated the development of procedures for 
manually managing trainees and resources- Our analysis indicated 
that instructors would be performing the following instructional 
management functions: 

0 administering and scoring tests; 

0 making lesson assignments; and, 

0 monitoring trainee progress. 

In addition to these management functions, the instructors would 
continue to provide tutorial and remedial aid for individual 
trainees. 

No special procedures were developed to aid in the administra- 
tion and evaluation of criterion reference checks (i.e., perfor- 
mance tebts) and written or oral tests qiven at the end of 
"knowledge" lessons. Scoring keys and tenplates were devised for 
hand processing both forms of the end-of-block knowledge test. 

Two charts were developed for instructor use in making lesson 
assignments. The first, the Lesson Flowchart, depicted a lesson 
network that specified acceptable alternate paths through Block X. 
The second, the Trainee Progress Flowchart, shown in Appendix D, 
contained a record of the lessons he had completed, the amount of 
time expanded in completing each lesson, and the lesson on which 
he was currently working. 

A trainee's performance on a criterion check, written test, 
or oral test given upon completion of a lesson was used by the 
instructors to determine whether he required Temedial instruction 
or was ready for a new assignment. If a trainee required remedial 
instruction, he was directed back to a prograimied text for 
"knowledge" lessons or to a videotape program for "skill" lessons. 
Upon cotr.pleting the remedial assignment he was retested. If his 
performance was not acceptable, he was given additional remedial 
aid. When his performance was acceptable, he was considered 
ready for a new assignment. When a trainee was judged ready for 
a new assignment, the Lesson Flowchart was examined to identify 
those lessons that he could take next. If the options were a 
"skill" and a "knowledge" lesson, the instructors checked to see 
if the appropriate resources (AV and test equipment) were avail- 
able for the "sMll" lesson. If the resources were immediately 
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available, the trainee was assigned to the workbench. If the resources 
were not inniediately available, the instructors chj^cked the Trainee 
Progress Flowchart to see if someone was close to tompletinq a "skill" 
lesson. If someone was going to be tested soon, the trainee awaiting 
a TOW assignment was told to use the photo book or a videotape program 
to preview the next lesson. If the resources were not imirodiately 
available and no one was going to be tested soon, the trainee was 
directed to a "knowledge" lesson. Lesson presentation sequences were 
limited to the options delineated in the Lesson Flowchart- Each 
trainee, of course, was required to complete all of the Block X les- 
sons - only order of completion was allowed to vary across trainees. 

In addition to being used in the prescriptive process, the 
Trainee Progress Flowchart was used to monitor the performance of 
individual trainees. A trainee's performance history in terms of 
the number of lessons conpleted and the amount of time expended com- 
pleting each of those lessons could be compared to similar data for 
his classmates. These comparisons allowed the instructors to identify 
trainees who were experiencing difficulty and to isolate those lessons 
on which the group was expending a great deal of time. Finally, the 
Flowchart summarized performance data in an easy-to-read format so 
that individual or group data could be easily captured for report gen- 
eration purposes. 

TASK 15: DELINEATE COMPUTER ADAPTION RATIONALE 

The instructional management functions mentioned In the preceding 
section are all computer-compatible. The first function, test scoring, 
serves as basis for the other two functions. The second function, 
making lesson assignments, calls for the application of decision rules 
to performance data to generate Instructional prescriptions. The 
third function, monitoring trainee progress, requires that the perfor- 
mance data be analyzed and manipulated so that performance and time on 
lesson is accessible on an individual, group, or class basis. The 
requirements for computerizing the management of Block X in an indivi- 
dualized, self-paced mode are described in the paragraphs that follow. 

Test Scoring - Computerized test scoring would require the develop- 
ment o?~programi~to score multiple choice, true/false and constructed- 
response tests as well as performance checklists. Performance check- 
lists would have to be standardized and made to resemble objectively- 
scored tests from a format standpoint. Constructed-response tests 
could not be scored unless there was a human interface between the 
tests themselves and the data input to the computer or the tests were 
read and scored in a CAI mode. 

Making Lesson Assignments - Additional programming would be 
required to enable the computer to compare trainee performance and 
time data (single lesson or cumulative lesson data) with validated 
instructional treatment data. A schedullng-resource allocation program 
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would provide infonnation regarding limited t^soyrces currently 
available and the forecasted demand for those limited resources. 
Decision rules would be input to the computer. These statements of 
logical and co?nputational operations, in conjunction with formatting 
sta tenants, would process performance and resource availability data 
in such a way as to output prescriptions. 

Monitoring Trainee Progress - The test scoring programs would 
require the addition of statements that would cause the performance 
and time data to be stored In files which permit a flexible retrieval 
capability. In addition to trainee performance and time data, infor- 
mation pertaining to instructional alternatives and previously 
encountered instructional treatments could be stored. A report gen- 
eration program would be constructed which would query the appropriate 
files and output performance, time, and instructional treatment data 
on a lesson or cumulative lesson basis for an individual trainee, 
selected trainees, or the entire group of trainees currently inter- 
acting with the system. 

StCTION V 

EVALUATION OF THE PROTOTYPE INSTRUCTIONAL MATERIALS 
AND SPECIFICATION OF MANHOURS EXPENDED AND CONVERSION COSTS 

The tasks that comprised the major activity in Phase IV and the 
order in which they were carried out were as follows: 

ACTIVITIES TASKS 

(Phase I) (1, 2, 3, 4, 5) 

(Phase II) (6, 7, 8, 9, 10, 11, 12) 

(Phase III) (13, 14, 15) 

16. Describe Formative Evaluation and 
Validation process 

Evaluation of the X 17. Evaluate the Prototype Instructional 

Prototype Materials -^^^ Materials 

18. Specify Manhours Expended and 
Conversion Costs 

Detailed descriptions of how these tasks were accomplished are present- 
ed in the pages that follow. 
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TASK 16: DESCRIBE FORMATIVE EVALUATION AND VALIDATION PROCESS 



Formative evaluation and validation of the Block X prototype self 
paced materials involved an iterative process of reviews, tryouts, 
instructor debriefings, and revisions. The process is depicted graph- 
ically in Figure 4. The initial step in the process involved having 
technical experts review the prototype lessons and suggest revisions. 
The prototypes were revised and 10 trainees then used them in an oper- 
ational classroom setting. Data such as trainee performance on 
criterion-reference checks and the end-of-block examination as well 
as training time per lesson and total training time and trainee att.- 
tudes toward the self-paced materials were utilized in developing a 
debriefing agenda for use with PME Master Instructors. The debriefing 
dealt with problems encountered in the smal 1 -group tryout and led to 
suggestions for revising prototype materials and manual management 
procedures prior to exposing large groups of trainees to the revised 
prototypes. The prototypes underwent a second revision and 35 
trainees used them in an operational classroom setting. Learning 
and attitudinal measures and training times were utilized 1n develop- 
ing a final debriefing agenda for use with PME Master Instructors. 
Suggested revisions were made and the lessons were then considered 
operatioftal . 
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FIGURE 4 BLOCK X FORMATIVE EVALUATION AND VALIDATION PROCESS 



TASK 17: EVALUATE THE PROTOTYPE INSTRUCTIONAL MATERIALS 

Fourteen trainees utilized the progranuned instruction lessons 
constructed for the Semiconductor Physics and Transistor Theory seg- 
ments of Block IV, and 12 trainees utilized the programmed instruction 
lesson constructed for the Logic Circuit segments of Block VI. The 
rtsults of these tryoucs are shown in Table 1. In general, the data 
indicate that the trainees were able to master the materials in much 
less time than was conventionally devoted to the topics in question. 
We concluded that our progranmed instruction format was appropriate 
for imparting knowledges of the type required in the PME course. 
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TABLE 1 

TRYOUT RESULTS - BLOCK IV & BLOCK VI SEGMENTS 





NO, OF 
TRAINEES 


CROUP 
TEST SCORE 


CONVENTIONAL 
COURSE HOURS 


X TIME TO a»<PLETE 


BLOCK IV 

SEMICONDUCTOR PHYSICS 


14 


83"i* 


6 HRS. 


2 HRS., Z^, MINS. 


TRANSISTOR THEORY 


14 


96":** 


6 HRS 


1 HR., 42 MINS. 


BLOCK VI 










LOGIC CIRCUITS 


12 


99^0*** 


9 HRS. 


3 HRS., 32 MINS. 



• 7 QUESTIONS ON END-OF-BLOCK EXAMlNATICW 
** 7 OUESTICWS ON END-OF-BLOCK EXAMINATIOM 
•••6 QUESTIONS ON END-OF-BLOCK EXAM1NATIC»4 



The 10 trainees who took part in the Block X small -group tryout 
completed the materials in 83% of the normal block time, thereby 
achieving a mean time savings of 17%. Sunmiary statistics for Block X 
small -group tryouts are shown in Table 2. As a group, the trainees 
successfully completed 269 out of 270 criterion- reference checks 
(27 checks/man) on the first try (overall man = 99%). They 
achieved a mean end-of-block examination score of 77.4%, with all 
trainees passing the end-of-block examination on the first try. 
Trainee attitudes toward the self-paced materials were assessed 
anonymously upon completion of the end-of-block examination using 
the 20-1 tern scale shown in Appendix E. All negative items were 
scored positively. A score of 60 would have been indicative of a 
completely neutral attitude toward the self-paced materials. The 
mean score was 78.4%, Indicating that trainee attitudes toward the 
self-paced materials wert; highly favorable. Test scores, attitude 
data, and time saved are shown in Appendix F. 



TABLE 2 

SMALL-CROUP TRYOUT RESULTS - BLOCK X 





NO- Of 1- 

TRA^Nrrsj^ ^^^^ ^^o^E 


CONVf NTJONAl 
COURSF HDi\RS 


X im 10 COMPLETL 


k TlWf SAVED 


" TlMf SAVED 


WAVEFORM ANAL rS»S 


^0 ?7.4* 


90 H«S. 


n Hf^^j , ?4 MINS. 


1^ hR$.,36 MtNS. 





'H'-lTrM END-OF -BLOCK EXAMINATION 
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The 35 trainees who took part in the large-group validation com- 
pleted the revised Block X materials in 69% of the normal block time, 
thereby achieving a mean time savings of 3]%. Simwiary statistics 
for Block X large-group validation are shown in Table 3. This was a 



TABLE 3 

LARGE-CROUP VALIDATION RESULTS - BLOCK X 





NO 0^ 5 


CONVtNTiONAl 
COURSE HOURS 


X 7!Mt TO COMPLETE 


J< Time SAyED 


TiMf SAVCP 


WAVE FOKM ANAl fSIS 


! 1 1 


00 MRS 


OJHRS., 24 MINS. 


2; MRS.. 36 MtNS. 
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definite improvement over the time savings achieved by the 10 trainees 
who took part in the small -group tryout. The improvement may have 
been due to revisions made in the prototype materials following small - 
group tryout, or the added experience of the instructors in adminis- 
tering self-paced instruction. Most probably it was due to some 
combination of these two factors. 

As a group, the trainees who took part in the large-group vali- 
dation successfully completed 937 out of 945 criterion-reference checks 
(27 checks/man) on the first try (overall mean = 99%). There was no 
difference between their performance and that of the trainees who took 
part in the small-group tryout. It should be noted that critericn- 
reference checks were conducted on an individual basis during the 
small -group tryout and large-group validation. In the conventionally- 
taught block, trainees work and are tested in pairs. The instructors 
felt that performance testing was much more rigorous and that they 
were better able to assess a trainee's competence when he was tested 
on an individual basis. Additionally, trainees were able to spend 
more time working with the equipment in the self-paced mode of 
instruction. 

The large-group validation trainees achieved a mean end-of-block 
examination score of 80.1%, with 32 of the 35 trainees passing on the 
first try. Their nrean examination score represented a slight improve- 
ment over that achieved by the trainees who took part in the small - 
group tT70uts. The three trainees who failed the end-of-block examin- 
ation on their first try subsequently passed an alternate form of the 
examination after completing their remedial assignments. The mean 
score of the 35 large-group validation trainees on the attitude scale 
was 77.4%, indicating that their attitude toward the self-paced mate- 
rials was highly favorable. Test scores, attitude data, and time 
saved are shown in Appendix G. 
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Background data were obtained on 47 trainees who received conven- 
tional instruction in Block X In nearly the same time frame as the 35 
valid.nion trainees. The data for the 35 validation trainees and the 
other 47 trainees were compared with respect to general and electronic 
AQE, and Block I, X, and total test scores (these data are shown in 
Appendix H). The differences were exceedingly small and did not favor 
either group consistently. This indicates that the validation trainees 
did not differ radically from their contemporaries trained convention- 
ally. 

TASK 18: SPECIFY MANWURS EXPENDED AND CONVERSION COSTS 

A total of 5515 manhours were expended in developing the individ- 
ualized instructional materials for selected segments of Blocks IV and 
VI and for Block X of the PME course. A total of 71 contact hours of 
instruction were produced (36 hours of programed text and 35 hours of 
mediated (AV) instruction). The total number of manhours expended on 
each task in the materials developnwnt process is shown in Table 4. 
The mean number of manhours expended per contact hour of instruction 
was 79. For prograiraned text« the mean number of manhours expended 
per contact hour was 65. For mediated (AV) instruction, the mean 
number of manhours expended per contact hour was 94. 

In addition to the manhours expended in implementing Block X In 
a self-paced mode, it is possible to enumerate the costs associated 
with converting the conventional classroom into a configuration capa- 
ble of supporting individualized instruction. A total of 15 indivi- 
dualized instruction stations were set up, 6 performance stations, 6 
study stations, and 3 preview-remediation stations. 

Each performance station contained all of the electronic test 
equipment appropriate for PME skill lessons. It also contained an 
Audiscan device, an A. B. Dick device, an equipment photo book, a 
set of AV scripts, and a full complement of Audiscan and A. B. Dick 
cassettes. Each performance station, then, was an independent unit - 
it contained all of the resources necessary to complete the skill 
lessons. The cost of setting up a performance station was as follows: 



Audiscan Device (1 § $280.00) $ 280.00 

Audiscan Cassettes (22 @ $20.00 each) 420.00 

A. B. Dick Device (1 9 $290,00) 290.00 

A. B. Dick Cassettes (8 0 $25.00 each) 200.00 

Equipment Photo Book (1 0 $90.00) 90.00 

AV Scripts (1 Set 13 $20.00) 20.00 

$1300.00 
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TABLE 4 



MANHOURS EXPENDED PER TASK 

p^,ase I MANHOURS EXPENDED 

1. Develop Segment Selection Criteria 20 

2. Review & Analyze Job Documents 80 

3. Isolate Trainee Characteristics 40 

4. Examine Traininq Environment 40 

5. Select Segments -4S- 

220 



Phase II 



6. Conduct Training Analysis 1140 

7. Review Learning Objectives 200 



8. Update Test Items 200 

9. " 

10. 

11. . 

12. Select Media 



Specify Sequence 120 

Specify Presentation Strategies 40 

Develop Media Selection Rationale 40 

Select Media , 4 0 



Phase III 

13. Prepare Materials 

Programmed Instruction 940 

Audio-Visual ^180 

14. Develop Management PUn 40 

15. Delineate Computer Adaptation Rationale 40 

2900 



Phase IV 

16. Describe Formative Evaluation Process 80 

17. Evaluate the Materials 555 

18. Specify Manhours Expended and Conversion Costs 80 



Total Manhours Expended 5615 
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The 6 perfonnance stations were supported by a performance "spares" 
station that had the following Inventory: 

Audi scan Device (1 9 $280.00) $ 280.00 

Audiscan Cassettes (9 sets 0 $420.00 each) 3780.00 

A. B. Dick Device (1 @ $290.00) 290.00 

A. B. Dick Cassettes (9 sets 9 $200.00 each) 18(K).00 

AV Scripts (9 sets ^ $20.00 each) 180.00 

$S350Cr 

There were no spares for the electronic test equipment. 

Each study station contained the 5 programined instruction (PI) 

texts appropriate for Block X. It also contained an equipment photo 

book and a set of AV scripts. The cost of setting up a study station 
was as follows: 



Five PI Texts ($50.00 each) $250.00 

Equipment Photo Book (1 @ $90.00) 90.00 
AV Scripts (1 set @ $20.00) 20.00 

Nine extra copies of each of the 5 PI texts were available for 
Individual study after normal classrown hours (Cost = $2250.00). 

Each preview-remediation station contained a videotape recorder/ 
player and monitor, a full complement of videotapes, an equipment 
photo book, and a set of AV scripts. The cost of setting up a preview- 
remediation station was as follows: 



Videotape Recorder/Player (1 @ $860.00) $ 860.00 

Videotape Monitor (1 p $335.00) 335.00 

Videotapes (26 © $24.00 each) 624.00 

Equipment Photo Book (1 5 $90.00) 90.00 

AV Scripts (1 set e $20.00) 20.00 

$1920.00 



.10 hardware or courseware spares were available for preview-remediation 
stations. 



The total cost of converting the conventional classroom into an 
Individ' alized instruction center was as fol lows: 



6 Performance Stations ($1300.00 each) $ 7800.00 

Spares Station ($6330.00) 6330.00 

6 Study Stations ($360 00 each) 2160.00 

Study Spares Station ($2250.00) 2250.00 

3 Pre view- Remediation ($1929.00 each) 5787.00 

Stations $24327.00 
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SECTION VI 



CONCLUSIONS AND REC0W1ENDATI0HS 

The reduction in training time achieved with the prototype self- 
paceU materials arques strongly for the feasibility of individualizing 
the PME course as part of the AIS. Achiev«nent as measured by 
criterion-reference checks and the end-of-block examination was well 
above the acceptable minimum, and trainee attitudes toward the fnate- 
rials were highly favorable. Most importantly, the trainees in the 
self-paced groups, unlike their counterparts in conventional instruc- 
tion, were able to work alone at performance stations. Each trainee 
was able to get "hands-on" experience with each of the major equip- 
ments, and performance testing was more rigorous and standardized to 
a greater degree than it was in conventional training. 

The model for designing, developing, and evaluating self -paced 
materials, described in Sections II, HI, IV, and V of this report, 
is currently being revised and extended to provide guidelines for AIS 
materials development efforts. Moreover, the model as describeJ 
would seem to be applicable to a wide variety of materials develop- 
ment efforts that have as their aim the design, development and 
evaluation of self-paced, multimedia training materials. Its point of 
departure is a reliable and thorough body of job tasks documentation. 
Consequently, those interested in applying the model in areas where 
job documentation does not exist or is suspect in terms of complete- 
ness or accuracy should realize that a significant amount of addi- 
tional "hard work" will be required. 

The prototype instructional materials developed in the cont ext 
of this contract are amenable to computer management for eventual 
use as part of the integrated AIS. When the test scoring, prescrip- 
tion, resource allocation, and report generation programs described 
in Section IV become operational. Block X materials and resources 
can be placed under computer control. 

No significant problems were encountered in any of the follow- 
iny areas: personnel requirements or procedures; equipment require- 
ments or procedures; instructional management; or integration of 
system elements. The following observations and recomn«ndatioris in 
these and related areas are, however, noted for your information. 

1. Technical coordination proved difficult and time-consuming. 
Initially, task analysis documentation, learning objectives, 
test items, programned texts, and scripts were mailed to 
the PMEL. This introduced a 4 to 7 day delay between 
completion of tasks by contractor personnel and the com- 
mencement of technical review by PME Master Instructors. 
Documentation and instructional materials were critiqued 
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and revisions negotiated via telephone conferences between 
the Master Instructors and contractor personi^l . Lack of 
face-to-face contact led to misunderstandings and some 
unnecessary duplication of effort. The eventual substitu- 
tion of site visits for the telephone conferences allevi- 
ated the problems associated with the nail -out and telephone 
critique procedures. 

For small-scale development efforts, we recommend frequent 
site visits for coordination, review, and revision of 
training documentation and prototype materials. For 
large-scale development efforts, we reconniend physical co- 
location of resource and contractor personnel. 

Two Instructors v«re required to handle the Instructional 
management functions during small-group tryout and large- 
group validation. The eventual autonation of several of 
those management functions will allow a single Instructor 
to handle an even greater number of trainees under AIS 
than he currently handles in conventional instruction. 

Manual managanent of self-paced instruction Is not a triv- 
ial matter. We recofwnend that provisions be made early in 
the materials design and development cycle for defining 
the new duties that the Instructor will assume under a 
self-pacing configuration. Procedures and tools for accomp- 
lishing new duties may then be proposed, refined, and 
solidified prior to field testing the prototype materials. 

A separate room was set aside for administration of the 
end-of-block examination. Normal classroom noises and 
other potential distractions would have worked an unneces- 
sary hardship on trainees being tested. 

We recommend that separate rooms be utilized for instruc- 
tion and testing In self-paced conf iqurations. If space 
limitations prohibit the use of two rooms, some practical 
means of isolating trainees who are being tested from 
those engaged in instructive activities should be devised 
(e.g., semi-enclosed testing carrels). 

The removal of electronic equipment from performance 
stations for repair and calibration caused delays and 
necessitated trainee re-routing during small-group tryout 
and large-group validation. 

We recommend acquiring a minimum of ore back-up unit for 
each niajor piece of equipment. If funds are limited. 
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equipment reliability, maintainability, and frequency of 
usage data can be utilized in selecting for purchase as 
spares those pieces of equipment that, in the event of 
breakdown, have the greatest adverse impact on trainee 
flow through the materials. 

End-of-block examination scores were not as high as 
expected. The absence of formal end-of-lesson tests 
almost certainly served to depress the end-of-block 
examination scores. Comprehensive tests should be given 
at the end of "knowledge" lessons. The possibility of 
inserting a review test at the end of the lesson to pro- 
vide trainees with feedback and the opportunity to reme- 
diate prior to taking the end-of-lesson test should also 
be considered. 

Instructors and trainees agreed that some of the longer 
progranmed instruction lessons were monotonous and boring. 

Programmed lessons that require more than one hour of 
uninterrupted reading should be divided and the new 
packages should not be presented sequentially. A brief 
review or introduction on a different presentation medium 
can be Inserted to avoid the boredom that results from 
reading for long periods of time. 

The decisions to use motion pictures for certain of the 
"skill" lessons was based primarily on our desire to 
implement more than one AV jrodiumln the feasibility 
study, and only secondarily on our belief that motion 
had special pedagogical value in the context of those 
lessons for which it was iitilized. 

In a production, rather than R&D, configuration the 
decision to use motion pictures should be based on a 
detailed analysis of lesson objectives, task characteris- 
tics, and a learning proficiency-production cost trade- 
off. 

Audi scan courseware failures, due to poor fllmstrip 
splicing, v«re numerous. 

Great care should be exercised in splicing the film- 
strips prior to inserting them into the Audiscan cart- 
ridge. Quality control procedures that require inspection 
of each spliced fllmstrip or a periodic spot check of a 
sample of spliced fllmstrips would insure a significant 
decrease in the number of classroom failures. A further 
recoitmendation calls for investigating the utility of 
sound/filmstrip devices that can be manually paced by 
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the trainee, fonvard or backward, without destroying the 
synchionicl ty of picture and sound. Finally, headsets 
and footswitches should beconie integral components of 
the presentation device. Headsets make for quieter 
classrooms, and footswitches allow trainees to control 
AV programs without affecting their performance on two- 
handed tasks. 

Trainees wef. not able to use the STOP feature of the 
A. B. Dick projector. In those instances when the STOP 
feature was used, a relay overheated and the film would 
not advance upon restart. 

If the A. B. Dick projector is to be used. It should be 
used as a continuous presentation device; i.e., the STOP 
feature should be deactivated or trainees should be told 
not to activate it. 
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APPENDIX A 
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SAMPLE TASK ANALYSIS 
FREQUENCY COMPARISON USING LISSAJOUS PATTERNS 



STEP 



Secure all loose clothing and remove all rings, watches, and 
other objects that could act as conductors of electricity and 
cause shock or electrocution. 

2- Insure that the TEK-545 POWER switch Is OFF and that TEK TYPE 
CA Plug-In Unit Is 1nstaTTi3~1n the TEK-545. 

3- Connect the TEK-545, HP-205AG, and HP-202A power cords to 
sources of 117 voit, 60 Hz power. 

4. Set the HP-205AG AMPLITUDE control to the 0 position and turn 
the POWER switch to ON. 

5. Set the HP-202A AMPLITUDE control fully CCW and turn the 
POWER switch to W. 

6. Set the TEK-545 front controls as follows: 



a. INTENSITY - CCW 

b. roCUS - Center 

c. ASTTfiNlATISM - Center 

d. HORIZONTAL POSITION - Center 

e. PSwER - On 

^- HORIZONTAL DISPLAY - Ext. Sweep 

Main Sweep 

a. TRIGGERING LEVEL - CW 

b. STABILITY (Red Knob) - CW 

c. TRIGGER SLOPE - + Ext 

d. TRIGS^RINfi W )'DE (Red Knob) - DC 

e. TIME/ CM - 10 Mil Usee 

f. MULTIPLIER - 5 

g. 5X MAGNIFIER - Off 



7. Set controls of CA Plug-In Unit as follows: 
Channel A 
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a. AC/DC SWITCH 

b. POLARI TY 

c. WTC^ POSITION 

d. VOLTS /CfT 

e. VARIAB LE' 

f. TwdTswitch 



AC 

Normal 
Center 
20 

Cal 1 bra ted 
A Only 
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Channel B 



b. 



c. 



a. 



e. 



d. 



ac/dc switch 
pSupity 

WSTOal position 



VOLTS/CM 
VARIABLE 



- AC 

- Normal 

- Center 

- 20 

- Calibrated 



8. Set the controls uf the HP-205AG Audio Signal Generator as 
follows: 



9. Advance HP-2056 AMPLITUDE CONTROL until the OUTPUT METER 
indicates 20V. 

10. Set the HP-202A, Low Frequency Function Generator as follows: 

a. AMPLITUDE control to 20 

b. ffi£OU£NCy dial to 3 

c. RANGE to X 10 

d. FllcTlON SELECTOR - Sine 

11. Turn the INTENSITY control clockwise until you see a trace on 
the screen. 

12. Set FOCUS , INTENSITY , and ASTIGMATISM for a sharp line. 

13. Position the trace near the center of the screen with the 
Channel A VERTICAL POSITION control and the HORIZONTAL POSITION 



14. Connect a cable from the HP-205AG OUTPUT jack to the TEK-54b 
Channel A INPUT jack, and the Main Sweep TRIGGER INPUT jack, 

15. Adjust for 4 CM of vertical deflection, on Channel A, with 
the VOLTS/CM and VARIABLE controls. 

16. Using the HORIZONTAL POSITION control center the straight 
vertical line horizontally. 

17. Disconnect the cable from the Channel A INPUT jack, 

18. Connect a cable from the HP-202A OUTPUT jack to the TEK-545 
EXT. SWEEP IN jack. 



a. FREQUENCY 

b. NO AtTiENUATION 

c. ll^EDANCE SWITCH 

d. LOAD SWiYCH 



- 60 Hz 



- "5000 

- On 



control . 
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19. Adjust the straight horizontal line for 4 CM of deflection 
using the EXT. SWEEP ATTEN. Control . It may be necessary to 
change the position of the ATTEN switch to obtain the 4 CM of 
deflection. 

20. Using the Channel A VERTICAL POSITION control center the 
trace vertically. 

21. Reconnect the cable from the HP-205AG OUTPUT jack to the 
Channel A INPUT jack. 

22. Stabilize the sweep using the HP-202A FINE FREQUENCY control. 

23. Record Frequency Ratio, where: 

Frequency Ratio « Vertical Deflection in CM: Horizongal 

Deflection in CM 

Frequency Ratio = . 

24. With the HP-205AG and the HP-202A at the following settings, 
record the Frequency Ratio. 



HP-205AG HP-202A Frequency 

Setting Setting Ratio 

a. 90 Hz 30 Hz 

b. 120 Hz 30 Hz 

c. 150 Hz 30 Hz 

d. 90 Hz 60 Hz _____ 



25. Turn the TEK-545, HP-205AG, and HP-202A POWER switches to OFF 
and disconnect all cables. 
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SAMPLE CONTENT ANALYSIS 
TEK-545 OSCILLOSCOPE: FRONT PANEL CONTROLS AND CONNECTORS 

Main Sweep 

a. Triggering Mode (Red Knob) 

(1) Five-position switch 

(2) Arranges trigger circuits for five kinds of 
tri ggering 

(a) HF SYNC 

(b) AUTOMATIC 
c) AC FAST 

(d) AC SLOW 

(e) DC 

b. TRIGGER SLOPE 

(1) Six-position switch 

(2) Selects source of trigger signal 

(a) EXT + or - 

(b) INT or - 

(c) LINE + or - 

c. TRIGGER INPUT 

(1) UHF Coax Connector 

(2) External Triggering Input 

d. STABILITY (Red Knob) 

1) Four possible positions or regions 

2) Adjusts sweep circuits for stable display 

(a) Free Running 

(b) Triggerable 

(c) Locked Out 

(d) Preset 

e. TRIGGERING LEVEL 

1 ) Potentiometer 

2) Determines the point, on the signal under test, whe 
the sweep will start or trigger. 
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f. TIME/CM 

(1) Eight-position switch - .1 ysec to 1 sec/cm 

(2) Detennines sweep speeds 

(3) Determines duration of trigger holdoff period 

g. MULTIPLIER 

(1) Six-position switch 

(2) Three positions determine sweep speeds with selected 

timing capacitor - XI, X2, and X5 

(3) Three positions, in red, allcw for continuous control 

of sweep speeds - 1-2.5, 2-5, and 5-12 

h. 5X MAGNIFIER 

il) Two-position switch (on/off) 
2) Change sweep speeds by factor of five 
3) Magnifies the presentation between the 4th and 6th 

i. + GATE 

(1) Connector 

(2) Supplies 20 volt positive pulse in synchronization 

with the main sweep 

j. SAWTOOTH 

(1) Connector 

(2) Supplies 150 volt positive going sawtooth 
Delaying Sweep 

a. STABILITY (Red Knob) 

(IJ Three regions 

;2) Adjustment of multivibrator 

13) Adjusts bias for recurrent or triggered sweep 

(a) Free Running 

(b) Triggerable 

(c) Locked Out 

b. TRIGGERING LEVEL 

(1) Potentionieter 

(2) Determines the point, on the signal under test, wnere 

a sweep will start or trigger 
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c. TIME/CM 

(1) Twelve-position switch 

(2) Selects twelve fixed sweep speeds - 2 ysec to 
10 msec/ cm 

d. LENGTH (Red Knob) 

(1) Sweep length control 

(2) Permits delaying sweep to be reverted imnwdiately 

to increase duty cycle 

(3) Normally left at 10 cm 

e. DELAY-TIME f^LTIPLIER 

(1) Ten-turn helical resistor (helipot) 

(2) Determines timing of delayed trigger 

f . RESET MAIN SWEEP 

(1 ) Pushbutton 

(2) Arms main sweep triggering circuit 

g. DELAYED TRIGGER 

(1) Connector 

(2) Supplies delayed trigger from main sweep or from 

delaying sweep 

h. + GATE 

(1) Connector 

(2) Supplies 20 volt positive pulse in synchronization 

with delaying sweep 

1. SLOPE + or 

(1) Toggle switch 

(2) Selects in or out-of-phase output for triggering 

sweep gating multivibrator 

(3) Selects polarity for external sweeps 

j. ATTEN. XI. XIO 

(1) Toggle switch 

(2) Inserts or bypasses 10:1 compensated attenuator in 

delaying- sweep trigger input or external circuits 
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k. EXT SWEEP ATTEN (Same as D.S. STABILITY) 

{}) Control 

(2) Gain control for horizontal anq)lif1er 

(3) Not used with internal sweeps 

1 . TRIGGER OR EXT SWEEP IN 
(1) Connector 

2) Input to horizontal anplifier 

3) 5X MAGNIFIER must be ON when used for EXT SWEEP 

Additional Controls 

a. HORIZONTAL DISPLAY 

1) Four-position switch 

2) Arranges sweep circuits for displays of: 

(a) Main Sweep Normal 

(b) Delaying Sweep 

(c) Main Sweep Delayed 

(d) EXT Sweep 

b. HORIZONTAL POSITION 

1 ) Potentiometer 

2) Positions trace 

c. VERNIER (Red Knob) 

1) Potentiometer 

2) Fine horizontal position control 
Other Outputs 

a. SQUARE-WAVE CALIBRATOR (Red Knob) 

1) Three-position switch 

2) Voltage divider to give either VOLTS or MILLIVOLTS 
output 

b. SQUARE-WAVE CALIBRATOR (Black Knob) 

(1) Nine-position switch 

(2) Provides nine accurate peak-to-peak voltages or 
millivolts position of the red knob. .2 to 
100 VOLT or MILLIVOLT 
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c. CAL OUT 

1) UHF connector 

2) Provides SQUARE-WAVE CALIBRATOR output voltages 

d. CAL OUT 

1) UHF connector 

2) Provides SQUARE-WAVE CALIBRATOR output voltages 

d. VERT SI6 OUT 

1 ) Connector 

2) Supplies a sample of the vertical -deflection signal 

e. 6.3V AC 

(1) Connector 

(2) Provides output of 6.3 VAC from oscilloscope power 

transformer 

iary Controls 

a. POWER 

(1) Toggle Switch 

(2) Power ON/OFF control 

b. FOCUS 

(1) Potentiometer 

(2) Controls voltage to CRT focusing grid 

c. INTENSITY 

1) Potentiometer 

2) Bias control of CRT control grid 

d. ASTIGMATISM 

1 ) Potentiometer 

2) Controls voltage to CRT astigmatism grid 

e. SCALE ILLUM 

(1) Po tent i one ter 

(2) Controls current through graticle lights 
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SAf«>LE LEARNING OBJECTIVE 



Given the appropriate manual and technical orders and test 
equipment, the trainees will make five frequency comparisons using 
Lissajous patterns, record each frequency ratio, and ccmmit no 
errors of practical significance. 

GIVEN 



AFM 127-101 

T.O. 33A1-13-73-1 

T.O. 31-1-141-1 through 18 

TEK-545 with TEK TYPE CA Plug-In Unit installed 

HP-205AG 

HP-202A 

Appropriate power cords 
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Date 



DIRECTIONS: Below are several statements about the course of instruc- 
tion you have just completed. Respond to each statement with how you 
felt while participating in this instruction. 

There are no right or wrong answers. 
Read each statement carefully and 
indicate how much you agree or 
disagree. 



2. 



I would like nwre instruction presented 
in this way. 

I learned more because these instruc* 
tional materials were available for me 
to use. 



3- This instruction was very boring. 

4. In view of the time allowed for 
learning, I felt that too much 
material was presented. 

5. I became easily discouraged with this 
type of instruction. 

6. There are too many distractions with 
this method of instruction. 

7. I felt that I wanted to do n\y best 
work while taking this instruction. 

8. This method of instruction makes 
learning too mechanical. 

9. This is a poor way for me to learn 
skills. 
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10. This method of instruction does not 
seen to be any better than other 
methods of Instruction. 

n. It was hard for me to follow the 
order of this instruction. 

12. I felt uncertain as to ray 
performance in the instruction, 

13. There was enough time to learn the 
material that was presented. 

14. I don't like this Instruction any 
better than other kinds I have had. 

15. This was a very good way to learn 
the material . 

16. While taking this instruction I 
felt challenged to do ny best work. 

17. Answers were given to the questions 
that I had about the material. 

18. I seemed to learn very slowly with 
this type of instruction. 

19 This type of instruction makes me 
want to work harder. 

20 I felt as if I had my own teacher 
while taking this instruction. 
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8EST m ^u;^iU^iL 



BLOCK X 
SMALL-GROUP TRYOUT RESULTS 



TRAINEE TEST SCORE DAYS SAVED AHITUDE 

1 92 4 84 

2 88 4 90 

3 84 2 76 

4 80 3 72 

5 76 1 72 

6 74 4 74 

7 70 3 81 

8 70 3 93 

9 70 2 70 
10 70 0 72 

I 77.4 2.6 78.4 

Range 70-92 0-4 70-93 
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BLOCK X 

LARGE-GROUP VALIDATION RESULTS 



TRAINEE 


TEST SCORE 


1 


76 


2 


66 




76 


3 


80 


4 


70 


5 


92 


6 


88 


7 


82 


8 


92 


9 


72 


10 


82 


n 


84 


12 


78 


13 


92 


14 


84 


15 


70 


16 


70 


17 


70 


18 


72 


19 


94 


20 
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21 


92 


22 


80 
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84 


24 


74 


25 


80 


26 


96 


CI 


70 
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82 


29 


80 


30 


82 


31 


92 


32 


78 


33 


68 




82 


34 


68 




88 


35 


84 




80.1 


Range 


66-96 



DAYS SAVED ATTITUDE 



5 


-- 


2 


92 


- 
6 


-- 
84 


3 


75 


4 


85 


5 


100 


3 


79 


3 


75 


3 


74 


2 


68 


5 


73 


6 


mm 


4 
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